Q^ . Carnot in the Information Age: 

o' 
o 

(N 



o '■ Discrete Symmetric Channels 



i> 



Ido Kanter^, Ori ShentaP, Hadar Efraim\ Nadav Yacov^ 

^ Department of Physics, Bar-Ilan University, Ramat-Gan, 52900 Israel 
^\| I ^ Center for Magnetic Recording Research, University of Cahfornia, San Diego 9500 

Oilman Drive, La Jolla, CA 92093, USA 

^' 

St , Abstract. Modeling communication channels as thermal systems results in 

Hamiltonians which are an explicit function of the temperature. The first two 
authors have recently generalized the second thermodynamic law to encompass 

jrt . systems with temperature-dependent energy levels, dQ = TdS+ < d£/dT > dT, where 

tr. ' < ■ > denotes averaging over the Boltzmann distribution, recomputing the mutual 

information and other main properties of the popular Gaussian channel. Here the 
mutual information for the binary symmetric channel as well as for the discrete 
symmetric channel consisting of 4 input/output (I/O) symbols is explicitly calculated 
i-rt ■ using the generalized second law of thermodynamics. For equiprobable I/O the mutual 

^ ' information of the examined channels has a very simple form, -jU{'j)\q, where U 

denotes the internal energy of the channel. We prove that this simple form of the 
mutual information governs the class of discrete memoryless symmetric communication 
channels with equiprobable I/O symbols. 
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1. Introduction 

The current scientific conception is that the theory of information is a creature of 
mathematics and has its own vitahty independent of the physical laws of nature [1]. 
The first two authors have recently proved |2l [3] that the principal quantity in the 
theory of information, the mutual information, can be reformulated as a consequence 
of the fundamental laws of nature - the laws of thermodynamics. This corollary was 
originally exemplified for the Gaussian noisy channel. 

The generic problem in information processing is the transmission of information 
over a noisy channel |H El E]. This central paradigm of information theory can be 
mathematically abstracted to having two random variables X and Y representing the 
desired information and its noisy replica, respectively. Noisy transmission can occur 
either via space from one geographical point to another, as happens in communications, 
or in time, for example, when sequentially writing and reading files from a hard disk in 
the computer. 

Mutual information, I{X\Y), quantifies the amount of information in common 
between two random variables and it is used to upper bound the attainable rate 
of information transferred across a channel. To put differently, mutual information 
measures the amount of information that can be obtained about one random variable 
(channel input X) by observing another (output Y). A basic property of the 
mutual information is that /(X; Y) = H{X) — H{X\Y), where H{-) is the information 
(Shannon) entropy |lj. It measures the amount of uncertainty in a random variable, 
indicating how easily data can be losslessly compressed. Hence knowing Y, we can save 
an average of /(X; Y) bits in encoding X compared to not knowing Y. 

As recently shown [21 [3], the modeling of the Gaussian channel as a thermal 
system requires the generalization of thermodynamics to include temperature-dependent 
Hamiltonians, as well as the redefinition of the notion of temperature. The generalized 
second thermodynamic law was proven to have the following form 

T T\ dT / ^ ^ 

where < ■ > denotes averaging over the Boltzmann distribution. 

The generalized second law of thermodynamics ([1]) has a clear physical 
interpretation. For simplicity, let us assume that an examined system is characterized 
by a comb of discrete energy levels SI, £2, . . ., see Figure la . The heat absorbed into 
the T-dependent system has the following dual effect: A first contribution of the heat, 
dU— < dS{X)/dT > dT, increases the temperature of the system. Figure lb, while the 
second contribution, < dS{X)/dT > dT, goes for shifting the energy comb. Figure Ic. 
However, the shift of the energy comb does not affect the entropy, since the occupation 
of each energy level remains the same, and the entropy is independent of the energy 
values which stand behind the labels S1,S2, . . .. The change in the entropy can be 
done only by moving part of the occupation of one tooth of the energy comb to the 
neighboring teeth. Figure lb. Hence, the effective heat contributing to the entropy is 
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Figure 1. A system consisting of a discrete energy comb (levels), e™, with the 
corresponding degeneracy r2(em) which increases with the energy. For the simplicity 
of presentation we assume that the system is occupying only one tooth of the comb, 
depicted by a red tooth in Figure la. As heat is absorbed into the system, the system 
can either increase the temperature by jumping to the next tooth of the comb, Figure 
lb, or shift the energy comb. Figure Ic. Note that only the jump to the next tooth. 
Figure lb, changes the entrpoy of the system, where in Figure Ic the entropy remains 
the same as in Figure la. 



dQ— < dS{X)/dT > dT, and this is the physical explanation to the generalized second 
law ([T]). A schematic picture of communication- heat-engine is depicted in Figure 2, 
where the heat is devoted to the change of the Hamiltonian (without altering the 
thermodynamic entropy) is denoted by the term 'working channel'. Note that for T- 
independent Hamiltonians the traditional picture of heat engine is recovered as well as 
the traditional second thermodynamic law [3, [U [9] . 

Using the generalized second law of thermodynamics, (1), one can show [21 [3] that 
the generalized expression for the mutual information is given by 



-7f/(7)l^ + 



I (f/(7,y)+7<^>.|.)rf7X 



(2) 



Note that this thermodynamic expression for the mutual information holds for any 
channel which can be described by a thermal system exhibiting quasi-static heat transfer. 
For the Gaussian channel with a standard Gaussian input and signal-to-noise ratio, snr, 
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Figure 2. A schematic communication-lieat-engine. H/C denote Hot/Cold 
temperatures. 
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Figure 3. The input/output transition probabilities for the binary symmetric channel 



one can show [21 [3] that the celebrated formula for the Shannon capacity pLj, is obtained 
from (2), I{X- Y) = \ log (1 + /?), where /? = snr. 



2. Binary Symmetric Channel 

We turn now to derive the mutual information for the archetypal discrete memoryless 
channel, the binary symmetric channel (BSC), with input x and output y. The input's 
prior distribution obeys P{x = 1) = P{x = —1) = 1/2 and the probability for a symbol 
to flip during the transmission is denoted by 6, P{y = ±\x = ^) = 6, see Figure 3. 
Hence, the conditional probabihty of the output given the input is 



1 — xy l-\-xy 

P{y\x) = 5^~(1 - 6)^2- = exp 



f ln(^) + ^ln(5(l-5)) 



(3) 



A comparison of the channel's a-posteriori probability distribution, given by Bayes' law 

P{X = x\Y = y) <x P{Y = y\X = x) (4) 

with the Boltzmann distribution law yields the following mapping of the energy 
and the inverse temperature, /3, of the equivalent thermal system 

E 

2 

1-6 



-f-^lnW-^)). 



/3 = ln. 



6 



(5) 



Note that since the inverse temperature, P{S), is positive, then 6 = 1/(1 + exp(/3)) < 
1/2 [ini [H]. This definition of 6 coincides with two limiting cases. For 6 = 1/2, 
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/9 = (T ^ oo), where for 5 = 0, /9 ^ oo (T = 0). Using < x\y >= y{l — 2S), one can 
find that the internal energy is given by 

Note that for the BSC specifically the internal energy is independent of y due to the 
binary nature of the I/O symbols. Similarly to the internal energy on can find 

^R^-^ ln(l+exp(/j)) , exp(/3) 

^dp ■'^'' 13 ^l + exp(/3)- ^> 

It is easy now to verify that the second term of the generalized mutual information, (2), 
vanishes 

U{P)+ < Pj^ >.\y- ^ + 1 + ,^p (^) - 0, (8) 

and the mutual information for the equiprobable input-output (I/O) BSC (which is also 
the Shannon capacity of the BSC in general) has a very simple form given explicitly by 

I = -lU{^)f, = P{1 -5)- ln(l + exp (/3)) + 1 = 

(1 - 6) ln(l - 6) + 6\n{6) + 1 = 1- H{6) . (9) 

Hence, the mutual information for the BSC is recovered using the generalized second 
thermodynamic law. Note that in contrary to the Gaussian channel, in the BSC case 
the second term of the mutual information, (2), vanishes, and the mutual information 
is proportional to the internal energy 

I = -lUi^)\l (10) 

3. 4-I/0 Symbols 

Is this simple thermodynamic form, (10), a coincidence of the BSC only or it occurs for 
the general class of discrete memoryless channels? To find the answer to this interesting 
question, we first turn to discuss in detail the case of 4-1/0 symbols. The input is 
represented by two binary units Xi and X2, and similarly the output units are yi and 1/2. 
We assume that the 4-input symbols are equiprobable. The conditional probability of 
P{yi,y2\xi,X2) obeys the following symmetry. The probability that both output units 
are equal to the input units, respectively, is 1 — 5, the probability that only one unit is 
equal is e5 (e < 1), and the probability that the two output units differ from the input 
is 5 — 2e6, Figure 4. Hence, 

^, I , ,_, _ l+xjvi 1+3:21/2 , _, itflBl 1:1^2^2 , _, illflBl i±f2y2 , _ „ _ l-3:iai l-^2i/2 ,,,, 

P{y\x) = (1 - 5)^ ^(e5)^ ^(e,5)^ ^{5 - 2e(5)^ ^, (11) 

and it is easy now to verify that 

P{X = x\Y = y) oc 

f Xil/i + X2I/2 , . 1-5 A , ^Wn ^\^3 2f. ^^ , XiX2yiy2 . A - S {1 - 2e) ^ 

Similarly to (5) one can find in this case 
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Figure 4. The input/output transition probabilities for a cliannel of 4-1/0 symbols. 



where 



£■ = --X^\y\ + 2^21/2 + xxX2y\y2) - - ln((l - (5)(5^) + jnfi^Wi, X2y2, e), (12) 
/ = {xiyi + X2y2) ln(l - 2e) - xiX2yiy2 ln( — 5—) - ln(e^(l - e)) , (13) 



(3 = H 



1-5 . 
5 ' 



(14) 



and the two hmiting cases are b = 3/4 and e = 1/3 (/3 = 0) and 5 = (/9 — ^ 00). Using 
the following conditional expectations 



< x^lym >= t/m(l -26 + 2e5) = |/„(1 - 25(1 - e)), 

< xiX2\yiy2 >= 1/11/2(1 - 4:65), 
one can find that the internal energy 

U{f3) = 

and again verify that 



5 + ^-i^^^^ + ^l-2...W + .n(l-2,l. 



dE 

u{p) + p< — > 



dl3 ' "1^" 



0. 



(15) 



(16) 



:i7) 



Hence the mutual information has again the simple form, (10), and is given explicitly 
by 

I = 2 + {I -5) ln(l -5)+ 2e5 \n{e5) + 5{l - 2e) ln((l - 2e)5), (18) 

which can be verified (by direct computation of the mutual information out of its 
definition) to have the correct form. The mutual information as a function of (5, e) 
is depicted in Figure 5, where the mutual information / = for 5 = 3/4 and e = 1/3. 
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Figure 5. The mutual information as a function of 5 and e for the discussed case of 
4-1/0 symbols. 



4. Equiprobable Discrete Memoryless Symmetric Channels 

For the general case of discrete memoryless symmetric channels with 2" equiprobable 
input symbols, the conditional probabilities has the following form 

P{yi, ...,yn\xi, ...,Xn) =Se{xiyi, ..., x„|/„), (19) 

and one can verify that similarly to (5) and (12) the energy is given now by 



E 



+ -nfn{.{xiyi}, {e(a;i|/i, ..., a:„|/„)}), 

where /„ is a function of the number of symbols only. Now it is clear that 

d\n{{l-5)5^''-^)d5 



(20) 



2-{U{(3)+ < (3— >,.|,) 



-<-l+Ul^^{l + Xkyk) > 



dS dj3^ 

where as in (5) and (12) P = ln((l — 6)/6) and dS/df] = —6{1 — 6). Hence 



dE 

2-{U{f3)+ < (3— >,.,,) 



< -1 + n^=i(l + Xkyk) > +2" - 1 - 2"5 = 

- < nLi(i + xm) > +2"(i - 5) = 0. (21) 

The identity to is a result of < 11^^^ (1 + Xkyk) > which is equal to the trace of the 
conditional probability, (19). Hence we proved that for a general memoryless symmetric 
channel consisting of 2" equiprobable I/O symbols 



Ui/3)+ < P— >.\y-- 



0. 



(22) 



-7f/(7)|[?, 
JE 

'Ip 

Note that the identity / = — 7f/(7) |o is in contrast to the case of a Gaussian channel 
with Bernoulli-1/2 or Gaussian inputs [21 [3]. 
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